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A workable theory of instruction appears to depend upon the development of 
measurement instruments, which operationally define concepts such as transfer, 
analysis, synthesis, and evaluation, to make possible empirical investigation of the 
relationships of methods of instruction and learner variables to the attainment of 
different types of educational objectives. This project developed 31-item sets in the 
physical sciences, and administered them to secondary school students. The sets of 
items followed generally the levels of understanding postulated in Bloom s Taxonomy 
of Education Objectives. The reproducibility coefficients were determined for each 
item set (.75 to .95). It was generally found that the difficulty of the "comprehension, 
"extrapolation," and "application" items was highly related to the nature of the 
principle involved. At the lower levels, item sets were typically very easy, while 
analysis" items were often too difficult for the majority of students. Item sets as 
constructed appear to be useful as research tools, but not for evaluation of 
instruction as it exists today. It is probably possible to construct other cumulative 
and hierarchical taxonomies of educational objectives in which the items increase in 
difficulty with each level and are highly reproducible and scalable. Sample item sets 
are appended. (Author /CJ) 
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INTRODUCTION 



The Importance of Structure 

A current trend in education la to examine the basic 
structure of the subject matter areas with an eye toward im- 
proving the understanding, retention, and applicability of the 
content and principles taught* This has been the general 
approach of the Biological Sciences Curriculum Study group, the 
American Association for the Advancement of Science group in the 
Elementary Science Curriculum Study, MIT's Physical Science 
Study group, and the Social Science Education Consortium* 

These groups are representative of the many different 
groups looking at the structure of the various content areas* 

To these can be added all of the various groups working on the 
new mathematics, devising different approaches to the teaching 
of foreign languages, and studying revisions in the English 
curriculum* It seems that every content area is being reviewed 
in an attempt to develop a structure which will result in 
students being able to perform the higher cognitive processes* 

Bruner^, in reporting on the Woods Hole Conference of the 
National Academy of Science, postulated four basic claims that 
have been made for teaching the fundamental structure of a 
subject claims which he says are in need of detailed study* 
These are as follows: 

1) That understanding fundamentals makes a subject more 
comprehensible* 

2) That the most basic thing that can be said about human 
memory* • • , is that unless detail is placed into a 
structural pattern, it is rapidly forgotten* 

3) That, an understanding of fundamental principles aid 
ideas, • • • , appears to be the main road to adequate 
"transfer of training*" 

4) That by constantly reexaming material taught in 
elementary and secondary schools for its fundamental 
character, one is able to narrow the gap between 
"advanced" knowledge and "elementary" knowledge* 



We have had in the past a great deal of repetitions and 
conflicting evidence as to the effectiveness of various instruct- 
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ional techniques (example ts. non-example; discovery va. 
verbalization; the effectiveness of various types of media; the 
effects of learning set; autocratic vs. democratic classroom 
procedures; lecture vs. discussion; rule-example vs. example- 
rule , etc.). It is the contention of the writer that a great 
deal of the Inconsistency of the findings can be attributed to 
the kind of measurement performed in attempting to assess the 
results of the experimental treatments. In most cases the 
instruments used are not reported with the findings f and no 
information is given regarding the nature of the items used. It 
is impossible to tell whether the items utilised measured 
anything other than the "knowledge” level mentioned in the 
Taxonomy of Educational Objectives: Cognitive Domain .* 

"Knowledge" according to the Taxonomy authors is considered to be 
recall or recognition without understanding. Added weight can 
perhaps be lent this position by the findings of the Taxonomy 
authors who state: 

Ve have also attempted to organise some of the 
literature on the growth) retention) and transfer of 
the different types of educational outcomes or behaviors. 

Here we find very little relevant research. For the 
most part) research on problems in retention) growth) 
and transfer has not been very specific with respect to 
the particular behavior involved. Thus # we are usually 
not able to determine from this research whether one 
kind of behavior is retained for a longer period of 
time than another or which kinds of educative ex- 
periences are most efficient in producing a particular 
kind of behavior. Many claims have been made for dif* 

Jtxent educational procedures) particularly in relation 
to permanence of learning; but seldom have these been 
buttressed by research findings. 

The authors of the Taxonomy ££ Educational Objectives have 
attempted to arrange the ordering of the classes of educational 
behaviors from simple to complex. The simple to complex 
ordering of the classes of the Taxonomy is based upon the notion 
that a series of simple behaviors combines to form more complex 
behaviors. The ordering implies that one must have "knowledge" 
before one can "comprehend"; that one must be able to "comprehend" 
before one can "apply"; that the "analysis" of a communication 
is more difficult than the "comprehension" or "application" of 
the rules or principles involved in the communication; that the 
"synthesis" of a communication requires a higher degree of under- 
standing than does the "analysis" of the "communication"; and 
finally) that "evaluation" of the communication in terms of 
internal or external evidences is the most complicated mental 
process. 



Brief definitions of the major categories of the Taxonomy 

follows 

Knowledge . Knowledge, as defined here, involves the recall of 
methods and processes, or the recall of a pattern, structure, 
or setting* For measurement purposes, the recall situation 
involves little more than bringing to mind the appropriate 
material* Although some alteration of the material may be 
required, this is a relatively minor part of the task* The 
knowledge objectives emphasise most the psychological 
processes of remembering* 

CgmDrehensi2&* This represents the lowest level of understanding 
It refers to a type of understanding or apprehension such that 
the individual knows what is being communicated without 
necessarily relating it to the other material or seeing its 
fullest implications* 

Application . Application involves the use of abstractions in 
particular and concrete situations* The abstractions may be 
in the form of general ideas, rules of procedures, or 
generalized methods. The abstractions may also be technical 
principles, ideas, and theories which must be remembered 
and applied* 

Analysis* The breakdown of a communication into its constituent 
elements or parts such that the relative hierarchy of ideas 
is made clear and/or the relations between the ideas ex— 
pressed are made explicit* Such analyses are intended to 
clarify the communication, to indicate how the communication 
is orgnized, and the way in which it manages to convey its 
effects, as well as its basis and arrangement* 

Svntheais. Synthesis is the putting together of elements and 
parts so as to form a whole. This involves the process of 
working with pieces, parts, elements, etc*, and arranging and 
combining them in auoh a way as to constitute a pattern or 
structure not clearly there before* 

Evaluation . Evaluation involves judgments about the value of 
material and methods for given purposes, quantitive and 
qualitative judgments about the extent to which material and 
methods satisfy criteria, and the use of a standard of 
appraisal* The criteria may be those determined ty the 
student or those which are given to him* 

Evidence is mounting that the cognitive process suggested 

fcy the are M erarchial and cumulative (Smiths $ Cox-^j 

Fur at 5; Dressed) at least through the application level. 



It is the contention of most authorities in the field that 
teachers should utilise a hypothesis testing approach to education. 
Tyler, ^ Travers, * McDonald, 9 and Bloom , *• to name a few, have 
suggested that the educational act should consist of (1) specifying 
the specific desired student behaviors, (2) designing learning 
experiences to achieve these behaviors and (3) evaluating to 
determine the extent to which these behaviors have been achieved. 

The hypothesis regarding the effectiveness of instruction is 
then dither accepted or rejected. 

The Taxonomr sL Educational Objectives: fagnltlTtt SSBftlA 

intended to provide a classification for the goals of our 
educational systems. It was designed to help teachers and 
administrators deal with curricular problems by providing a model 
for the analysis of such cognitive educational outcomes as 
remembering, thinking, and problem solving. 

Measurement in our schools is at the present time largely 
limited to the lowest Taxonomy category of "knowledge." 

According to the Taxonomr . the "knowledge" category is not 
considered a form of understanding. 

It is likely that in our schools, where the importance of any 
objective seems to be determined largely by our ability to measure 
it, grade conscious students memorize and give us back exactly 
what we measure: "knowledge" without understanding. The effect- 

iveness of the Taxonomy for measuring the higher mental process 
involved. 

In view of the evidence supporting the existence of a 
hierarchial and cumulative continuum of cognitive processes we should 
re-examine our position on the measurement of student achievement 
in the schools. The entire concept of content validity should be re- 
examined in the light of the scalability of the cognitive process. 

The cognitive levels suggested by the Taxonomy &£ 

Objectives indicate that the teacher should be concerned not only 
with the content validity of his test, but should also be concerned 
with the level of understanding measured by the test. The 
cumulative and hierarchial structuring of cognitive processes 
suggest that the teacher should, in fact, be concerned with how 
far the learner is along the way toward being able to achieve the 
"application," " analysi," "synthesis," or evaluation," levels. 

The instructional procedure used for the student who is at the 
"knowledge" level is probably quite different from the instruct- 
ional procedure that is effective in getting the student, who 
comprehends a principle, to position where he can apply it in 
different situations. The instructional task involved in moving 
the student from the ability to apply a principle to a position 
where he can analyze a theory or structure in tens 
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of internal and external evidences imply greatly different 
instructional techniques* 

The cumulative and hierarchial structuring of the cognitive 
processes further suggests that ve should develop cumulative and 
hierarchial sets of items that lead up to and measure the 
behavioral objective at the level specified* The sets of items 
should follow the logical structuring of the snbject matter, and 
build to the cognitive levels specified by the educational 
objectives* Thus, if we are going to develop courses that attempt 
to build a logical theory or structure, we should construct sets 
of test items that will measure t^e important steps along the 
way to analysis and evaluation of’ the structure* 

Rationale for the Study 

Cumulative and hierarchial sets of items are badly needed 
in 111 subject matter areas* The item sets must be developed so 
that we can: 

1* Diagnose more accurately a student *s learning difficulties 
to determine how far he has come toward achieving the 
specified criterion level stated in the educational 
objectives* 

2* Determine how effective various teaching methods are for 
bringing a student to higher levels of understanding* 

3* Investigate the relationship of intelligence to the 
various cognitive levels of achievement* 

4* Investigate the relationship of retention to the level 
of taxonomic achievement acquired by the student* 

5. Investigate the part played by reading in achieving a 
high score on the sets of items - particularly the 
higher level items of analysis and evaluation* 

6* Determine whether or not the higher levels of the 

Taxonomy (analysis and evaluation), are truly different 
cognitive processes from those used in "comprehension 11 
and "application,” and look into the problem of teaching 
for the cognitive processes of "analysis” and "evaluation* 

7. Relate motivation to the level of understanding achieved. 

8* Gather more evidence regarding the age and grade levels 
at which the various cognitive functions cm be performed* 
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9. Examine the characteristics of learners as they relate 
to the level of understanding achieved* 

General Objectives g£ $fe£ Project 

This project is of importance to curriculum improvement at 
all grade levels. It would seem that the instruments suggested 
in the project would be of importance in helping to determine 
what can be taught at the various grade levels as well as giving 
us information regarding the degree of understanding that we can 
logically expect to obtain. The measurement devices should also 
be of some use in helping teachers evaluate the effectiveness of 
different instructional techniques for attaining the various 
Taxonomy, levels of understanding. The scalable test items can 
also be used to provide information regarding functions at any 
given grade level. 

Reaaaroh Related && g Taxonomic Measurement Cognitive 
Structure and Samples g£ Materials Previously Developed 

Research work concerned with the measurement of levels of 
understanding has largely developed using Bloom's XSSSRfiflE SL 
Educational Objectives ; Cognitive SfigglB as a theory base. The 
experimental approaches utilised in attempting to validate the 
Taxonomy have generally followed two patterns. The first group 
of experimenters generally attempted to validate the Taxonomy's 
structure by the process of classifying already constructed 
items. The experimenters then looked at the average difficulty 
level of the items in each major Taxonomy class. The rationale 
followed considered that if any category inoludes the processes 
of all lower-order categories, plus a unique component, then the 
mean score should decrease as the level of category increases. 
This inverse relationship between level of the Taxonomy category 
and difficulty level has held up consistently. 

This group of experimenters has also looked at item and 
sub-test intereorrelations in an attempt to determine the degree 
to which these correlations formed a Guttman simplex. If the 
slmplicial structure is satisfied, the correlation matrix will 
be such that the largest values will be along the upper-left, 
lower-right diagonal, followed by the next larger values in the 
adjacent diagonals and the smallest values in the upper-right 
mm! lower-left corners of the matrix. This approach has not 
been satisfactory because of the way differences in item 
difficulty level effect phi coefficients. This problem is 
central to the theme of Kropp, Stoker and Baslow'a article,® 
"The Validation of the Taxonomy & ttdn«atim>ai Objectives." A 
similar finding had also been reported earlier by Guttman'. 
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A second group of researchers interested in the scalability 
of the cognitive processes suggested by the Ift22S29Z» approached 
the problem by constructing items at the various Ttfogoig levels • 
These items were developed logically out of a particular concepts 
For example. Smith and Paterson' developed a twenty-item multiple- 
choice test consisting of four items relating to each of five 
principles. The items did not measure cognitive processes above 
the sub-class of ’•extrapolation” in the Taxonomy hierarchy. 

Five four-option multiple-choice items measuring "knowledge-of- 
pr inc iples** were written (one item for each of five principles 
related to the teaching-learning situation). The five knowledge- 
of -principle” items required only that the student be able to 
recall a principle— not that he understand it. The principles 
or relationships measured were: 

1) The principle of selective perception 

2) The relationship of norms to test interpretation 

3) The relationship of environment to intelligence test 
performance 

4) The relationship of set to problem solving 

5) The relationship of test-retest reliability to relative 
placement on the distribution curves 

After the completion of the five " knowledge-of-principle" 
items, five multiple-choice items that required that the students 
* extrapolate" from a specific principle were constructed. The 
stem of the "extrapolation" items consisted of a hypothetical 
situation. The student was then asked to specify what action should 
be taken in view of his understanding of a particular principle. 

It war found extremely difficult if not impossible to write 
multiple-choice items in the behavioral sciences that required 
the student to respond to a hypothetical situation without 
specifying which principle or theory in terms of which he was to 
respond. These items nearly always contained ambiguity stemming 
from the current state of psychological theory. Thus, it was not 
possible to write items which fulfilled the TMSBSBBte criteria 
for "application" items. 

Five other multiple-choice items were written in an attempt 
to measure the "interpretation" of basic communications dealing 
with the meaning of the five principles. The stem of the items 
included a basic communication designed to convey an understanding 
of the principle involved. The student was then asked to identify 
and comprehend the major ideas which were included in the 
communication as well as to understand their interrelationships. 



The final five items were designed to measure "knowledge” 
of the terminology necessary for "comprehension" and "extrapo- 
lation" from the five principles. The items which measure "know- 
ledge of terminology" are characteristic of the typical multiple- 
choice items used by instructors. The terms measured were those 
most relevant to the understanding of the principles to he developed. 

In order to prevent one item from giving clues to the other 
items involving the same principle it was necessary to order the 
items and place them on transparencies. The order in which the 
items involving a particular principle were presented is as 
follows: 

1) "Extrapolation" from the principle or relationship 

2) "Knowledge" of the principle or relationship 

3) "Knowledge" of terminology involved in the principle 

or relationship 

4) "Interpretation" of a communication concerned with the 

understanding of the principle or relationship 

The item intercorrelations for each of the five educational 
psychology concepts at the four Taxonomy levels were adjusted for 
the differences in difficulty levels of the items by the use of 
phi-to-phi max. ratios. The phi-to-phi max. ratios were then 
converted to Z scores and averaged over each of the four TlfffflPlT 
classes. The Z scores reported are low and generally positive. 

They do not confirm a slmple-to-complex continuum of the Taxonomy . 

The above statistical procedure, while it did tend to take 
into account differences in difficulty levels, and did raise the 
correlation coefficients, still suffers greatly from the problem 
previously mentioned. 

Perhaps the most significant finding of this study is the 
lack of relationship between items dealing with the same principle 
but requiring different cognitive processes. An examination of 
the phi-to-phi max. ratio shows a range from .07 to .40 of the 
total possible phi correlation between "knowledge of the 
principle" and "extrapolation" from the principle, the .40 phi- 
to-phi max. ratio is in all probability spuriously high because 
one of the items was too easy and did not function properly. 

The items listed below build logically to "extrapolation" from 
the principle. The highest phi-to-phi max. ratio found in this 
group of test items is .39* Note that this group of items does 
not Include the item measuring "comprehension" (i.e., translation 
or interpretation) of the principle. This is because it was 
generally found that items written at this level vary with all of 
the factors that affect reading difficulty and thus, in general 
do not scale. 
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A "set" is 



a. a facilitating group of behaviors related to a 
stimulus situation. 

b. an inhibiting group of behaviors related to a 
stimulus situation. 

c. a non-functional group of behaviors related to a 
stimulus situation. 

*d. a predisposition to view a problem situation in a 
given way. 

2. Which of the following best explains the relationship 
of •bets* to the solving of intellectual problems? 

a. A set enables the individual to transfer large 
bodies of facilitating information to all related 
problem situations. 

b. Sets pertain to skill learning, but not to problem 
solving. 

c. Sets are inconsistent with the notion of stimulus- 
response problem solving. 

*d. Sets narrow perception, and thus may interfere with 
problem solving. 

3. An arithmetic teacher wishes to have her students learn 
to recognize and solve the following types of problems: 

1. Q is what percent of B? 

2. X percent of A is what number? 

3. B is X percent of what number? 

According to your knowledge concerning the formation of 
sets, the teacher should: 

a. Have the individuals work until they discover the 
relationship. 

b. Work a series of the first type, and then a series 
of the second type, followed by a series of the 
third type of problem. 

*c. Present all types of problems in each exercise. 

d. Spend a week on each separate type of problem, and 
then repeat this series several times during the year. 

The highest correlation between these items was .39 between 
"knowledge” of term and "knowledge" of principle (the others were 
considerably lower). However, a Goodenough reproducibility 
coefficient of .95 was found among the three items when they were 
administered to 106 educational psychology students. The coefficient 
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